Studying how a robot can learn to communicate with a person provides insight into how communication might be learned in general. Deep models of dialog and communicative intent typically rely on modeling the internal state of the speakers—states that are unobservable by a learning robot. This paper considers how communication can be framed to be learnable from experience. In particular, we describe how an agent might learn to communicate by building on three foundational capabilities, namely 1) an observable signal of satisfied intent (a smile), 2) the ability to imitate perceived actions, and 3) perceptual referents for discourse items. Early simulation results show that an agent can learn some basic communication skills from these foundations.

In this paper, we propose a novel human-robot-interaction framework for the purpose of rapid visual scene understanding. The task of the robot is to correctly enumerate how many separate objects there are in the scene and to describe them in terms of their attributes. Our approach builds on top of a state-of-the-art 3D segmentation method segmenting stereo reconstructed point clouds into object hypotheses and combines it with a natural dialog system. By putting a `human in the loop', the robot gains knowledge about ambiguous situations beyond its own resolution. Specifically, we are introducing an entropy-based system to spot the poorest object hypotheses and query the user for arbitration. Based on the information obtained from the human-to-robot dialog, the scene segmentation can be re-seeded and thereby improved. We present experimental results on real data that show an improved segmentation performance compared to segmentation without interaction.

In this paper we present a model of language contextualization for spatially situated dialogue systems including service robots. The contextualization model addresses the problem of location sensitivity in language understanding for human-robot interaction. Our model is based on the application of situation-sensitive contextualization functions to a dialogue move's semantic roles — both for the resolution of specified content and the augmentation of empty roles in cases of ellipsis. Unlike the previous use of default values, this methodology provides a context-dependent discourse process which reduces unnecessary artificial clarificatory statements. We detail this model and report on a number of user studies conducted with a simulated robotic system based on this model.